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TRAINING METACOGNITIVE SKILLS FOR PROBLEM SOLVING 


High-level commanders and executives are expert problem solvers, but we 
understand very little of the nature of their expertise. We know that the executives 
have considerable knowledge and skill that enable them to perform their tasks in an 
expert manner, but we understand very little about the nature of this knowledge and 
these skills. The skills in particular are poorly conceptualized. We know they are not 
the kind of behavioral skills that psychologists have traditionally studied, such as riding 
a bicycle or shooting a rifle. Instead, these skills are “cognitive" skills; they involve 
manipulation and use of the elements of domain knowledge for some purpose, to 
some end. 

A skill is defined as an ability to do something well, and a cognitive skill is thus 
defined as an ability to perform a cognitive task well. A cognitive task is one in which 
successful performance depends primarily on the possession and skillful manipulation 
of information and knowledge; the product of a cognitive task is usually cognitive as 
well — an idea, a plan, a decision, a solution to a problem. In the domain of executive 
performance by Army commanders, problem solving may be considered the generic 
cognitive task, encompassing the formal mission-planning tasks and the less-clearly 
defined decision-making tasks of Army command and control. 

In addition to cognitive skills, executives seem to possess even higher-level 
skills that enable them to use their cognitive skills effectively. Called “metacognitive” 
skills, these are defined as abilities to monitor and direct the operation of cognitive 
skills to obtain the greatest possible success. Consider the example of what is 
perhaps the greatest cognitive skill of humans, the ability to construct sentences to 
convey meaning (that is, language). Metacognitive skills in the language domain 
include the use of one's knowledge of grammar or the lexicon to form more effective 
sentences, monitoring the response of the listener to diagnose communication 
success, and knowing when a picture is worth a thousand words. 

THEORIES OF METACOGNITIVE SKILLS 

Piaget’s stages of intellectual development. Jean Piaget has a theory of 
intellectual development that can be extended to the adult years. Infants are said to be 
at a relatively primitive, sensory-motor stage of intellectual development that, with the 
onset of speech, becomes a conceptual-symbolic stage called preoperational (Flavell, 
1963). Around the age of six, children enter the stage of concrete operations, in which 
they can apply operations (mental routines) to transform information in some way — 
adding two numbers to get a third, placing all red objects in the same pile. Around the 
age of 12, children begin the final stage of intellectual development called formal 
operations, in which they can apply mental routines to abstract material. For example, 
an adolescent can solve a problem like "If a suitcase can eat four rocks in one day, 
how many can it eat in two days?" Younger children cannot imagine a suitcase that 
eats rocks, so they will refuse to answer the question; they cannot disregard the 
content of the problem (its concrete aspects) and reason in a purely hypothetical way 
(using the form, or formal aspects, of the problem). 

The advent of formal reasoning creates an interest in form, that is, adolescents 
become fascinated by the formal structures and processes of thought. They think 
about thinking, which is a good definition of metacognition. One of the products of 
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Piaget’s theory is a body of research on metacognition, much of it on memory or 
“metamemory”: this research will be discussed below. 

Metamemory. Piaget’s theory led to research on metacognitive skills in 
memory, or metamemory. The specific topic in which we are most interested is 
problem solving or what we might call meta-reasoning. Nevertheless, the research on 
metamemory is of interest, not only because it represents a productive approach to 
metacognitive skills, but also because it has been a developmental approach, which 
offers clues to the development of metacognitive skills in general. 

John Flavell distinguishes between two broad areas of metamemory skills 
(Flavell & Wellman, 1977). The first is sensitivity to the need for planful memory. At 
first, the need for planful memory may be explicitly stated by a teacher or parent, who 
may instruct the child to remember something. Later the child may apply the 
metamemory skills spontaneously, knowing by now that one can prepare for later 
retrieval, that there is a difference between information processing for later recall and 
other cognitive processing of information. The second broad area of metamemory is 
knowledge of variables that affect memory performance. These variables include 
person variables (some people have better memories; people are likely to forget 
information learned under emotional stress), task variables (meaningless information 
is harder to remember), and strategy variables (rehearsal is a good mnemonic 
strategy). For example, children learn that if one variable, say task difficulty, is high, 
predicting poor memory, they must compensate with another variable, allocating more 
study time. The child may test memory and then concentrate rehearsal on the 
unlearned items. 
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Figure 1. A theory of metamemory. (From Nelson & Narens, 1994) 
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Nelson's theory of metamemory, although limited to one kind of cognition, is 
another framework for our analysis (Nelson & Narens, 1990; 1994). Figure 1 depicts 
the stages of memory — acquisition, retention, and retrieval — and gives examples of 
both the monitoring and control functions of metamemory skills. The monitoring 
functions determine the control functions to be activated. For example, suppose 
people are asked to learn some material to a certain criterion — "Learn this list of CIA 
agents perfectly, then destroy the list" — and monitor list acquisition until they make a 
metacognitive "judgment of learning" (JOL) or experience a "feeling of knowing" (FOK). 
If the JOL and FOK indicate that more study is required, control functions are activated; 
the kind of learning strategy is selected (rote memorization, use of mnemonic devices), 
and study time is allocated to the individual items on the list, according to the 
metacognitively perceived need. In a retrieval task, FOK and one's confidence in 
retrieved answers (CRA) determines search strategy and, ultimately, termination of the 
search. 

Adult stages of metacognitive development. Piaget's stages have been 
extended to the adult years by Schaie and Geiwitz (1982). The development of formal 
operations suggests that metacognitive structures and strategies can be applied to 
direct cognition toward problem solution, a skill that continues to develop throughout 
adulthood. The adult stages of intellectual development reflect a general increase in 
metacognitive skills, which underlie adult application and use of knowledge, rather 
than increases in cognitive skills, which underlie childhood acquisition of basic 
knowledge. The first adult stage, which occurs in young adulthood, is temporal 
monitoring, which represents the application of intelligence in situations that have 
profound consequences for achieving long-term-goals (involving decisions about 
career and marriage). Temporal monitoring is a kind of quality control process applied 
to problem-solving when the solutions must be integrated into a life plan that extends 
far into the future. It is similar to skills used by Army commanders when they prepare a 
synchronization matrix for the various Battlefield Operating Systems in a mission plan. 

A second major application of intellect in adulthood occurs in the second adult 
stage, called social monitoring. Typically this stage develops when a family is 
established, and the individual must begin monitoring not only his or her own 
behavior, but also that of spouse and offspring. Similar extensions of monitoring skills 
are required, as responsibilities for others are acquired on the job and in the 
community. Social monitoring includes temporal monitoring of a group of people who 
are all working toward the same end; not only must their activities be synchronized for 
maximum effectiveness, but metacognitive skills such as resource allocation and the 
efficient division of labor among group members become primary determinants of 
group performance. In Gardner's theory of multiple talents, social-monitoring skills fall 
into the category of personal Intelligence (Gardner, 1983), which includes the ability to 
take another person's perspective in a training situation; Anne Sullivan, the teacher of 
Helen Keller, is assumed to have been high in personal-social intelligence and skills. 

In the military domain, social monitoring is a set of metacognitive skills that will serve a 
commander of a combined-arms unit well. 

A third adult stage of intellectual development we call executive monitoring. 
Many individuals' responsibilities become exceedingly complex. They become 
presidents of business firms, deans of academic institutions, officials of churches, or 
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commanders of divisions or corps. As such, they need to understand how an 
organization works: the structure and the dynamic forces, who answers to whom, and 
for what purpose. They must monitor organizational activities not only on a temporal 
dimension (past, present, and future) but also up and down the hierarchy that defines 
the organization. Executive monitors must know the plans and intentions of superiors, 
and they must devise a structure for monitoring and controlling the implementation of 
policies at the lower levels of responsibility. 

Metacomponents of intelligence. The role of metacognition in 
problem solving has been investigated by Sternberg (1984, 1985) in support of 
his triarchic theory of intelligence. Sternberg has focused on the executive- 
process aspect of metacognitive skills, that is, the ability to organize, sequence, 
and monitor cognitive processes for maximum effectiveness. In the 
componential subtheory, three types of components (intellectual processes) are 
defined, one of which is metacomponents. “Metacomponents are higher-order 
executive processes used in planning, monitoring, and evaluating one's 
problem solving" (Sternberg, 1988, p. 132). In Sternberg's (1985) early writing, 
he listed seven prominent metacomponents: 

1. deciding just what the problem is 

2. selecting lower-order components to solve the problem 

3. selecting information representations 

4. selecting strategies for combining lower-order components 

5. deciding how to allocate attentional resources 

6. monitoring the solution: what has been done, what is being done, 
what still needs to be done 

7. monitoring feedback: altering behavior on basis of feedback: "How am 

_ I doing?" ____ 

in later works, Sternberg has focused on four metacognitive skills that are 
valuable in problem-solving tasks (Davidson, Deuser, & Sternberg, 1994). These are 
1) Identifying and Defining the Problem, that is, how to recognize that there is a 
problem to be solve; 2) Representing the Problem, that is, how to figure out what 
exactly the problem is; 3) Planning How to Proceed, which, together with 4) Evaluating 
Your Performance, helps you to understand how to reach a solution. These are 
relatively well-documented metacognitive skills that we will return to, in later sections, 
since they appear to have good construct validity and, when applied in a problem¬ 
solving task, promise significant benefit. 

With this sampling of theoretical approaches to metacognition, we turn now to a 
sampling of research results. This research provides data relevant to the value of 
metacognitive skills: Do they make a difference in performance in problem-solving 
tasks? In some studies, another question addressed is how the metacognitive skills 
are trained and assessed. 

METACOGNITIVE SKILLS AND PROBLEM-SOLVING PERFORMANCE 

There is not a large literature on training metacognitive skills and then 
evaluating the benefits on problem-solving performance. In general, the more 




common study seeks to determine if good problem solvers have more existing 
metacognitive skills than poor problem solvers. In general, the good problem solvers 
do have more of these skills, both as children and adults (Metcalfe & Shimamura, 
1994). In the domain of electronic trouble shooting, the most proficient technicians 
used more complex and more accurate mental models than the less competent 
electricians (Gitomer, 1988). More complex and more accurate mental models were 
also used by successful subjects in another study of troubleshooting abilities, in 
computer hardware problems; these subjects also used strategies to focus data- 
gathering activities (Reed & Johnson, 1993). 

I have personally developed hundreds of task-based proficiency tests for 
mechanics, electricians, and instrument/control technicians in the nuclear power 
industry. We discovered that, especially in the diagnostic tasks, the proficient 
technicians were those who had metacognitive skills to guide their investigations and, 
therefore, we had to assess these skills in our proficiency tests (Geiwitz, Spiker, & 
Harris, 1988; Spiker & Geiwitz, 1989). We then revised the training program for a 
subgroup of these technicians involved in ultrasonic testing of pipes for cracks. The 
good inspectors, we had learned, were those that developed a strategy of hypothesis 
testing when they recording a suspicious signal. They maneuvered their instruments 
to gather data relevant to competing hypotheses of crack vs. no_crack, or as is 
common in power plants, crack vs. weld_seam. So we inserted a one-hour segment 
on hypothesis testing into the 40-hour nuclear-power NDE (nondestructive evaluation) 
training courses. By this change, in 1 of 40 training hours, we were able to increase 
the frequency of successful detections by over 30% (Harris, 1992). This is a 
remarkable excess of benefit over cost that shows the promise of training personnel in 
metacognitive skills. In certain cases, it can result in an astounding increase in 
problem-solving performance. 

Nisbett's work on teaching reasoning is directly relevant to our project (Nisbett, 
Fong, Lehman, and Cheng, 1987). Psychology has shown that formal, deductive logic 
cannot be taught: it has also shown that humans rarely use formal logic in reasoning. 
What they do use in reasoning - mental models, causal schemas, and statistical 
heuristics -- is effective and can be taught. For example, educated laypeople use an 
intuitive version of the law of large numbers as a heuristic to aid them in reasoning 
about probabilistic events with uncertainties. The heuristic is, roughly, "Larger 
samples are required when generalizing about populations that are more variable 
than average." Thus, more observations of a baseball hitter are required to estimate 
his batting ability than of a neurosurgeon, whose surgical performances are less 
variable. These heuristics can be taught to others, as can similarly useful mental 
models and schemas. The interesting discovery of Nisbett's group, from our point of 
view, was that these metacognitive skills (using models strategically) cannot be trained 
by knowledge transfer alone, that is, the subjects must be given practical examples to 
solve using the skills: without repeated practice, no training benefit ensues. In our 
terms, Nisbett is teaching the procedural knowledge (the how to knowledge) along 
with the semantic knowledge (understanding the principles and heuristics). The 
procedural knowledge is crucial for training. We encountered a similar need for 
procedure-training when we tried to develop a course to teach supervisors how to 
construct performance tests of mechanical and electrical-skills. The supervisors 
soaked up test theory with ease and could repeat it flawlessly, if queried. But they 
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could not use the theory to actually construct tests until we taught them the step-by- 
step procedures for applying principles to real life situations (Geiwitz, 1992). 

TRAINING METACOGNITIVE SKILLS: 

A CASE STUDY 

In a most impressive implementation of a training program for metacognitive 
skills, Reif, Larkin, and their associates have developed training modules for science 
education, modules that emphasize and teach the metacognitive skills required for 
problem solving in college physics (Reif & Larkin, 1991). Their research began with 
an assessment of current educational difficulties. To summarize a considerable body 
of research, college physics students seem to "know" the concepts of physics, such as 
velocity and acceleration, but they cannot use this knowledge to solve even simple 
problems in that domain. For example, Heller and Reif (1984) found that Berkeley 
students (who had completed a basic physics course with a grade of B or better) could 
solve correctly only about 35% of typical textbook physics problems of the kind 
repeatedly encountered in their course. Other studies revealed that students' 
problem-solving methods were primitive and inadequate to the task (Larkin, 1982). 

KNOWLEDGE BASE AND PROBLEM-SOLVING STRATEGY 

Using a cognitive science approach, Reif and Larkin constructed instructional 
interventions for students' difficulties in scientific problem solving. They tried to identify 
the declarative and procedural knowledge required, especially the procedural, often 
the source of difficulty. The students did not know (were not taught) howto apply the 
concepts they had learned. Although most of the interventions were empirically based 
on experiments, theoretically the interventions turned out to be, in essence, training of 
metacognitive skills. For example, students were taught how to describe a problem so 
that the representation facilitates a solution (Heller & Reif, 1984). They were 
encouraged to formulate plans and strategies for problem solving. They were taught 
to recognize errors, especially when concepts had both an everyday and a scientific 
definition, with subtle differences that could lead to errors — "acceleration" is an 
example. Specific methods of concept application were also included in the 
metacognitive curriculum. In sum, the students were taught the process of problem 
solving: how to make judicious decisions, select appropriate methods, avoid dead 
ends, and recover from mistakes. 

The Reif-Larkin model for scientific problem solving consists of a problem¬ 
solving strategy together with a knowledge base, which provides the scientific 
information in the relevant domain (currently physics, specifically mechanics). The 
knowledge base is organized hierarchically, with central ideas described qualitatively 
at the highest level of the network and quantitative elaborations at lower levels. The 
principles in the knowledge base are accompanied by auxiliary information specifying 
when they are valid and/or likely to be useful. 

The problem-solving strategy is based on studies of expert problem solvers 
(Larkin, 1982) and on the strategies that worked in pilot implementation among 
college physics students (Heller & Reif, 1982). Like the knowledge base, the strategy 
is organized hierarchically. At the highest level, the strategy defines three 
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subprocesses; describing the problem, constructing a solution, and checking the 
solution. (Problem description is one of our metacognitive skills, and, as we shall see, 
the skills involved in the other two subprocesses are also largely metacognitive.) 

Lower in the hierarchical network, the main subprocesses are further 
decomposed. Problem description is decomposed into “problem summary* and 
"technical description." The problem summary Is a summary of available information 
about the situation, including the goal of the problem-solving endeavor. The technical 
description redescribes the problem in terms of properties of, and interactions among, 
the objects by using the relevant portions of the knowledge base. Its purpose is to 
facilitate application of the knowledge base to construct a solution. In mechanics, the 
technical description had two stages; 1) an interaction description, which uses a 
diagram (often from the problem summary) to identify all the interactions between 
systems: and 2) a system description, a diagram for each system of interest. Each 
diagram describes the motion of the system (in terms of position, velocity, and 
acceleration) and the interactions of this system (in terms of forces upon it). The 
diagrams also allow internal checks, e.g., the consistency between acceleration and 
total force. Arrows indicate vector quantities and direction, and equations indicate 
information about magnitudes. 

The second main subprocess, constructing a solution, involves decomposing 
the original problem into subproblems, each of which are solved in turn, ultimately 
leading to a solution of the original problem. Each solution requires the identification 
of alternatives and the judicious choice of one of these alternatives. Alternatives are 
explicitly stated in the knowledge base and strategy. Choice is based on an estimate 
of the utility of the consequences of that choice; if the choice proves unsatisfactory, one 
can backtrack to select one of the other alternatives. 

The actual construction of a solution specifies two complementary methods. 

The first method is to apply a central principle from the knowledge base (e.g., a law of 
mechanics) to a system in the problem with a convenient description (e.g., along a 
particular direction, if vectors are involved). This method simplifies a potentially 
complex choice by specifying three sets of concepts — a principle, a system, and a 
description — each of which provides only a small number of possible options, as 
suggested by the knowledge base and technical description. The second method 
involves removing unknown quantities in the relationships developed in the first 
method, by combining relationships (algebraically combining formulas to eliminate 
unknowns). The two methods can be reapplied as many times as necessary, until the 
problem is solved. 

The third main subprocess is an evaluation of the solution, to determine if it is 
clear, correct, and otherwise satisfactory. One of the primary checks is consistency, 
that is, the solution must be consistent with other known facts about the system in 
question, including the general principles of mechanics that apply to its behavior. 
Common mistakes can also be evaluated, to see if their influence in this particular 
solution has been substantial or negligible. 
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TRAINING PHYSICS STUDENTS 


In 1992, Reif and Larkin (1993) taught a section of the introductory physics 
course at Carnegie-Mellon University, with 56 randomly selected students. Other 
students took the regularly-offered course. On four common questions in the final 
exams of the two groups, the experimental section performed at a significantly higher 
level than the conventional section; see Figure 2. On the inclined plane problem (top 
left), 70 percent of the experimental students made no errors (or made only minor 
arithmetic errors): the regular students made major errors in all but 10 percent of 
observations! On other questions, the regular students fared somewhat better, but the 
experimental students retained their substantial edge, in addition to greater success 
in problem solving, the experimental students did not make the mistakes common in 
the conventional classroom, and they did not persist in erroneous reasoning as the 
conventional students did. In sum, the metacognitive skills training program was a big 
success. (Carnegie Mellon University asked Reif and Larkin to take over all of the 
introductory physics course, which they have done; they have lost their control group, 
but the size of the experimental group will allow more fine-grained statistical analysis.) 
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Figure 2. Percentage of students in regular and special sections who answered 
correctly the four questions on the final examination. 
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STRUCTURE AND CONTENT OF THE METACOGNITIVE TRAINING 

The basic structure of the Reif-Larkin approach has been described above and 
is summarized in Table 1. 


Table 1 

Components of the Reif-Larkin Training for 
Enhanced Problem-Solving Performance 


KNOWLEDGE BASE 


PROBLEM-SOLVING STRATEGY: 
problem description 

PROBLEM-SOLVING STRATEGY: 
construction of solution 


PROBLEM-SOLVING STRATEGY: 
evaluation of solution 


• concepts and principles 

• “operational* definitions 

• organized hierarchically 

• describes when principles to be applied 

• qualitative problem summary 

• technical (theoretical) description 

• decomposition into subproblems 

• application of relevant knowledge 

• application of relevant quantitative 
formulas 

• consistency 

• common mistakes 


Knowledge Base 

To illustrate the design of the domain knowledge base, I will describe the 
definitions and interpretations of the concept of acceleration contained in the textbook 
(Reif, 1993a). Acceleration is defined in Chapter 4 on Velocity and Acceleration, 
where motion is discussed in terms of the change of position of an object over time. 
After the discussion of velocity, acceleration, like all important new concepts, is 
introduced qualitatively, as a means of describing how rapidly the velocity of an object 
changes. This change may be in the magnitude of the velocity (the common sense 
definition of acceleration), but it may also be in the direction of the velocity, with no 
change in magnitude. Thus, “if a car travels along a curved road with constant speed, 
its velocity ... changes because the direction of its velocity changes* (Reif, 1993a, p. 
31). This description is supported by vector diagrams, as shown in Figure 3. 

To describe the rapidity with which velocity 
changes, the concept of average acceleration is 
defined as the change in velocity between time 1 and 
time 2 divided by the change in time (t2-t1). Average 
acceleration is easy to understand, but it is 
not very useful, because velocity can change in Figure 3. Vector diagram, 

several different ways within a lengthy time interval. What we want is a very short time 
interval or, more precisely, we want to define acceleration at an instant: a = dv/dt, in 
simple terms of the calculus. 
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Reif then presents the “defining method," that is, the things one must observe 
and/or do to identify acceleration in a particular case, as shown in Figure 4. The 
defining method is the "operational definition" of the concept. This definition is then 
supplemented with other knowledge about the concept, including the situations or 
conditions in which the definition is valid, the vector properties of the concept, 
warnings about common errors in interpretation, the units of acceleration, and what is 
required for a complete specification of the concept. 


Finding aeetieration ® dv/dt) 

(1) Original velocity Identify the velocity of the particle at the time t of 
interest 

(2) New velocity Identify the velocity of the particle at an infinitesimally 
later time t'. 

(3) Change of velocity dv. Fmd the velocity change dv= v' - v of the 
particle during the infinitesimal time interval dt = t* • t 

(4) Acceleration's. Fmd the ratio a=dv/dt [This is the “acceleration of 
the particle at the time dt is infinitesimally smalL] 

(5) Check. Qieck that the time t' has been chosen sufficiently close to t (i.e., 
that dt is sufficiendy small) that the value obtained for the acceleration 
would be unaffected by a closer choice. If this is not the case, repeat the 
method with a time t' chosen closer to L 





Figure 4. Defining method for interpreting the concept of acceleration. 
PROBLEM-SOLVING STRATEGIES 

To teach the problem-solving strategies, Reif (1993b) constructed a workbook, 
which is described in the textbook as "the primary learning aid." The workbook is 
designed to actively engage the student in interpreting and applying newly acquired 
knowledge. Whereas the textbook contains declarative knowledge, the workbook 
contains the critical procedural knowledge — what one actually does to interprei a 
concept in a particular instance. 

Problem description. How one describes the problem, as we have often 
suggested, is half the battle; Reif and Larkin teach an explicit method for describing 
any problem so as to facilitate a solution. The method involves two stages: 1) a basic 
problem description that clearly specifies the problem situation and goals; and 2) a 
more theoretical description in terms of relevant concepts and principles. The first 
stage translates the problem as found into a diagram of objects and their relationships, 
with useful symbols, time sequences, and other relevant information. The goal state of 
the problem-solving endeavor is also stated clearly and explicitly. 

In the second stage of problem description, concepts and principles from the 
knowledge base (textbook) are used to describe the mass of objects, the motion of 
objects, and especially all forces on the system of interest. Heller and Reif (1984) 
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specify an explicit procedure for generating this theoretical description, a procedure 
comprising the following steps: 

1) identification of the particular entities that should be described 

2) application of special concepts from the knowledge base to 
describe these entities 

3) exploitation of particular properties of these concepts 

4) application of particular principles from the knowledge base to 
check that the description is self-consistent and correct 

In physics, specifically in mechanics, the "particular entities that should be 
described" (step 1) are particles or systems of many particles. Particles or systems 
relevant to the problem are those about which information is desired and those that 
interact with such entities. The "special concepts" (step 2) are of two kinds. One, used 
to describe individual particles, includes concepts to describe the intrinsic 
characteristics of particles (e.g., mass) and other concepts to describe the motion of 
particles (e.g., acceleration). The second kind of concept is used to describe the 
interaction between particles (e.g., force). One should draw a motion diagram, 
describing the position, velocity, and acceleration of the relevant particles, and a force 
diagram, describing the external forces on the particles. The important properties of 
the special concepts (step 3) include various interaction laws that specify how the 
concepts describing interactions are related to the concepts describing motion, for 
example, how forces are related to acceleration. Some of these interactions are short- 
range, that is, they are substantial only when interacting particles are touching, and 
others (like gravity) may be substantial even with particles separated by some 
distance. In practice, step 3 prescribes the evaluation of forces exerted by each object 
that touches the relevant system, plus the long-range force of gravity (interaction of the 
earth with the relevant system). Finally, the knowledge base includes "motion 
principles" that specify how the motion of particles changes as a result of interactions 
among them (e.g., F = ma, which relates the acceleration of a particle to the sum of 
forces exerted on it). This principle should be used to check the descriptions of motion 
and interaction (step 4); for example, the acceleration of each particle must have the 
same direction as the total force on it. Step 4 cannot simply direct the student to 
ensure that motion and interaction descriptions are consistent with a specified 
principle. It is necessary also to indicate how one can determine such consistency. 
(The specific instructions for each step are contained in Appendix A.) 

When this procedure was used in an experiment, physics-student subjects 
solved an average of 2.75 difficult problems out of 3.00, compared to 0.63 for control 
subjects who had had relevant physics courses but who were given no guidance 
(Heller & Reif, 1984). Several of the experimental-group subjects were surprised at 
the efficacy of the procedure, stating with some amazement that the problems 
suddenly became very easy to solve! 

Solution construction. The strategy that Reif and Larkin teach to construct a 
solution to the problem is essentially "divide and conquer." The problem is divided 
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into a series of subproblems, each of which is conquered in turn, until the problem as a 
whole is solved. In most cases, two stages exist for solving the subproblem: 1) 
application of a principle (here, a mechanics law) to the system under consideration, 
and 2) eliminating unknown and unwanted quantities from the relations specified in 
step 1, by algebra. Figure 5 shows a skilled description of a problem, and Figure 6 
shows’the three-part solution. The solution depends upon the application of three 
basic relationships describing motion with constant acceleration: 

• relation (velocity, time): vx - vxo = axt 

• relation (displacement, time): Dx = vxot + 1/2 axt^ 

• relation (velocity, displacement): vx^ - vxo^ = 2axDx 


V2 = 0 @f 
T= 1.20 s. h 



[g 

balooay 


L= 13.0 m 




4 grauiid 


®_>@ = T = 1.20 s 
(D-»<3); a* = - g 

Known: T.L 

finals: (a) Vi s ? 
(b) h = ? 

(0 V3 = ? 


Situation diagram with time eequence 
indicated by eucceeeiw im p ort an t instants 

(D, (2), O.[(D -»Q> indicates the time 

elapsed between kistants O and Q>.] 

®: Sail leaves hand wth speed Vy 
®: Sail at highest point, height h. 

(3): Sail at ground, distance L below hand. 

The balfs acceleration is e^fual to ^ durkig 
the entire time (!>-»($. There is thus only 
one distret time Interval 

Velocl^ at highest point is zero shce height 
there momentah^ neither increases nor 
decreases, ie., doee not change. 

Wanted unknmm quantities are doub^ 
underlined in the diagram. 


Figure 5. How to describe a problem. 
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(a) (Fmdvj.) Rd(vel, t), ®->®, up 
Vx-Vxo“axt 
0-VI «-gT 


Heed to reiete veloetUee end known ttne T. 
Hence eppty ral(vel.t}. 

MoCkm ^leeetibed relative to upwartl t 
direction. 


vi « gT I 

vi -( 9.80 m/s^) ( 1.2 s) 
vi » 11.8 m/s 


( 1 ) 


Result in terms of symbols. 


a) 


Kiumerlcel result. 


(b) (Find h.) Rel(disp. t). up 

Dx«Vxot+ 58x12 


By(l): 


h - v,T -5 gT2 « (gT)T - 5 gT2 
b - ‘jgTi 


b-|( 9 . 80 iii/s 2 )( 1 . 2 s )2 
h s 7.1 m I 


( 3 ) 

(4) 


(c) (Find V3.) Rel(vd. diq)), ®-»®, down 


Vx^“Vxo2 = 2axDx 
V 32 _v, 2 « 2 gL 

V 3 - Vvi^ * 2gL 


( 5 ) 


vj -V( 11.8 in/s )2 + 2 ( 9.80 m/s 2 ) ( 13.0 m) 
V 3 » 19.9 m/s 




Need to relate u^ard displacement h to 
the known elape^ time T. Hence apply 
rel(disp. t). 


Result In terms of symbols. 


Numerical result. 

Need to relate velocities to displacement, 
without ben^ interested ki required time. 
Hence apply rel(vel. disp). 

Since we have decided to measure 
quantiUas downward, a^ ■ 4^ and D, ■ ‘*'1- 


Figure L How to construct a solution. 

The construction of a solution is less well specified by Reif and Larkin than 
other aspects of problem solving, e.g., problem description. Instead of a detailed 
procedure, Reif and Larkin desaibe the gei^ral procedure, as above, and rely on 
numerous examples, to be worked out by the student using the general procedure, 
Nevertheless, the details of the solution-construction process need more careful 
description, if metacognitive skills are to be trained effectively. 
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Solution evaluation. "The initial solution of a problem is rarely free of errors 
or other deficiencies. Hence any solution must be regarded as provisional until it has 
been checked and improved" (Reif, 1993a, p. 59). Reif presents the following 
questions, "useful to detect deficiencies in the solutions": 

• Have the goals of the problem-solving task been attained? (Has 
all wanted information been found?) 

• Is the solution well-specified? 

“ Are answers expressed in terms of known quantities? 

- Are units specified? 

- Are both magnitudes and directions of vectors specified? 

• Is the solution self-consistent? 

- Are units in equations consistent? 

“ Are signs (or directions) on both sides of an equation 
consistent? 

• Is the solution consistent with other information outside the 
problem? 

" Are values sensible (e.g., consistent with known 
magnitudes)? 

- Are answers consistent with special cases (e.g., extreme or 

simple cases)? 

- Are answers consistent with known dependencies (e.g., with 

knowledge of how quantities increase or decrease)? 

- Are answers consistent with those obtained by other solution 

methods? 

• Is the solution optimal? 

- Are answers and solution as clear and simple as possible? 

- Is the answer a general algebraic expression rather than a 
mere number? 

Like the construction of the solution, the solution check is phrased in general 
terms that make little sense to the students until they have worked through numerous 
examples. While practice is not a bad way to train these metacognitive skills, more 
detail should be given at some point, perhaps following the practice. 


The Reif and Larkin program is encouraging to those of us trying to develop 
training methods in other domains. As we have seen, metacognitive skills seem to 
enhance problem-solving performance, often to an unexpectedly high level. Nisbett, 
Reif, and others have shown us that these skills can be taught. We turn now to the 
question of which skills should be taught? Which metacognitive skills are most 
important for problem solving? 

METACOGNITIVE SKILLS 

Metacognitive skills are defined as abilities to monitor and direct the 
operation of cognitive skills to obtain the greatest possible success. A cognitive skill, 
in turn, is defined as the manipulation and use of the elements of domain knowledge 
for some purpose, to some end. In short, a cognitive skill enhances the performance 
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of a coanitive task. In the domain of executive performance by Army commanders, 
oroblem solving may be considered the generic cognitive task, encompassing the 
formal mission-planning tasks and the less-clearly defined decision-making tasks of 
Army command and control. 

Here from various sources and insights, is a list of ten metacognitive skills that 
have showri promise as candidates for training, if enhanced problem solving is our 

goal: 

Detection of a probiem. Recognizing the existence of a problem sounds 
easy enough, but in fact it is a skill that varies considerably among people, and it is 
highly correlated with intelligence and creativity (Sternberg, 1988). Somehow the 
individual must monitor the discrepancies between the current state and the goal st^e, 
noting a problem when the discrepancies exceed a certain value. Intelligent problem- 
solvers not only recognize that a problem exists, they are also ^tter at identifying the 
critical problems in a domain; in the words of one researcher, they have good taste in 
problems (Zuckerman, 1983). 

Representation of a problem. Once expert problem solvers have 
recognized the problem, they define the problem in a way that makes the problem 
soluSe (Sternberg, 1988). They represent the problem mentally (or in a computer 
program) in a form that is close to optimal for problem solution. This is an extremely 
valilable skill, perhaps the single most valuable of the metacognitive skills: How an 
individual states the problem is a prime determinant of success in solving it. 


Selection of a problem-solving method. There are many ways to solve 
oroblems. Good problem solvers know many methods, and they have the abi ity to 
select wisely, choosing appropriate procedures for solving the particular problems of 

the specific domain. 

Strategic application of problem-solving methods. Good problem 
solvers have strategies for solving the problem. They apply a potentially effective 
method constantly monitoring the changes in problem state that the method produces 
to see if a solution has occurred. They know what they will try next, and why. 
Strategies and methods in problem solving have much the same relationship as 
strategies and tactics in command and control. 

Evaluation of solution candidates. Like the wargame evaluation of the 
three Courses of Action selected in the mission-planning process, good problem 
solvers evaluate potential solutions, to see if the discrepancy between goal state and 
current state has been reduced. 

Recognition of errors. Good problem solvers spot errors more guickly and 
more accurately than poor problem solvers. Common errors that result from cognitive 
biases and misapplied heuristics of the sort studied by Tversky and Kahneman (1974) 
are anticipated and guarded against. 

Resource allocation. Good problem solvers, when they identify a problem, 
can allocate their problem-solving resources to create the most advantageous 
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environment for the solution to the problem. If the problem requires memory, for 
example, they know how long it will take to memorize the material, and they allocate 
the time accordingly. 

Temporal monitoring. Temporal monitoring includes the effective and 
strategic allocation of time resources, but it also includes the monitoring function, to 
see if the solution is developing “according to schedule.* Successful managers are 
noted for their ability to maximize the effective use of their time (Bray & Howard, 1981). 
In complex problems, many resources must be synchronized for maximum impact. 

Social monitoring. Problem solving in a social context — that is, most 
problem solving — is different from the same activity in isolation. Good problem 
solvers allocate human resources wisely, and they try to establish a social 
environment in which the group can function effectively. This means, among other 
things, they have to take personalities Into account, watch for conflicts, and moderate 
disputes. To manage effectively they must have sensitivity and understanding of other 
peoples' perspectives and goals. 

The social-emotional aspects of leadership became more important in 
American businesses as big corporations changed from family-owned enterprises 
controlled by autocratic individuals to publicly owned corporations led by committees. 
Psychologists were brought in to advise corporations on how best to solve problems In 
groups (Geiwitz, 1980). It was soon discovered that lack of knowledge and logic was 
not the chief impediment to effective group solutions; interpersonal relationships were 
much more crucial. One member of a committee would suggest a perfectly logical 
solution to a problem, but the group would reject it, because they disliked him or her. 
Thus, the original groups of executives brought together to learn how to solve 
problems — called “training groups* or “T-groups“ — were soon supplanted by 
"sensitivity T-groups" and, later, “encounter groups." Members of these groups 
learned how to recognize the emotional reactions their actions provoked in other 
people, usually through interaction and interpretation. 

Executive monitoring. Executives have key positions in a hierarchical 
network of individuals. To be effective problem solvers, they must understand their 
position in the network: their relationship with higher authorities, their relationship with 
subordinates, their relationship with peers. Executive monitoring is more than 
temporal and social monitoring, although it includes these lesser skills. In one study, 
for example, the major difference between young executives and older, more 
experienced executives was the greater ability of the older executives to market for the 
company (Schaie & Geiwitz, 1982). Marketing is a very high level skill, involving 
knowledge of what the company is trying to do, good perception of the needs of a 
potential customer, and a good sense of the company capabilities to solves certain 
kinds of problems. It usually develops slowly over the lifespan, and many executives 
never become accomplished at business development. 
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A CONCEPTUAL MODEL OF METACOGNITIVE SKILLS 


To develop a model of how metacognitive skills augment and facilitate the 
cognitive skill of problem solving, we need first a model of the problem-solving task. 
Complete, theory-bound models of problem solving do not exist, but the general 
stages and principles have been described by many; the Command Estimate is based 
on such descriptions. Problems are “initial states,” their solutions are “goal states,” 
and problem solving methods are means for transforming the initial state into a goal 
state (Newell & Simon, 1972). Problem solving in the business world (therein called 
management) has been described by Kepner and Tregoe (1965) and Plunkett and 
Hale (1982) in similar terms. The Command Estimate is another general description of 
the problem solving process. After analysis of the mission and the orders, relevant 
information about the situation (including the terrain of the battlefield) and the enemy is 
collected, in effect identifying and describing the problem and the goal state (the 
mission objectives). A number of actions that might solve the problem are described 
— Courses of Actions (COAs). Each COA is played out, step by step, in a wargame 
technique and evaluated in terms of several mission objectives. One COA is 
recommended, but ail are briefed to the commander, who makes the final decision. 

The Command Estimate, therefore, describes in more detail Step 4 above, suggesting 
that the problem solver generate options, evaluate these options, and then choose the 
option with the greatest apparent likelihood of success. These skills, however, may be 
better considered metacognitive. 

TECHNICAL MONITORING AND CONTROL 

Our conceptual model of metacognitive skills focuses on those skills that 
facilitate the technical activities in problem solving. The technical aspects of problem 
solving comprise the purely formal operations designed to identify, represent, and 
solve the problem. In addition to the technical aspects, there are temporal, social, and 
organizational aspects of the problem-solving process, no less important in many 
cases; we will discuss these aspects in a later section. 

Figure 7 tries to align the metacognitive skills relevant to monitoring and 
controlling the technical process of problem solving. In its present evolution, the 
conceptual model of technical skills identifies seven major capabilities: 

• detecting problems 

• representing problems 

• planning strategies for solving problems 

• applying the selected problem-solving method 

• implementing the solution 

• monitoring the solution 

• evaluating the solution 
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Figure 7. A model of metacognitive skills influencing technical problem solving. 


Of these seven metacognitive skills, strategy planning is the most complex, as 
broken down into more detail in Figure 8. Among the metacognitive skills involved in 
planning, effective and efficient allocation of resources is one that seems to be 
required at several points in the problem-solving process. A military commander, for 
example, will allocate the G2 staff to intelligence gathering and assessment, while the 
G3 staff begins the application of the problem-solving method (in this case, the method 
of the Command Estimate). The planner must also determine the most effective and 
efficient knowledge-acquisition technique, matching the knowledge characteristics of 
the problem domain with a technique for eliciting that knowledge from a particular 
source (Geiwitz, Komell, & McCloskey, 1991). The technique must then be applied 
properly to gather information. 

Once situation assessment has proceeded to the point of identifying the 
particular kind of problem being considered, a problem-solving method is selected 
with skills similar to those in selecting the knowledge-acquisition technique, matching 
the method to the problem type (Bylander & Chandrasekaran, 1988). Two common 
problem types are shown in Figure 8: "Caused" problems are the type for which the 
Kepner-Tregoe methods are appropriate, that is, something is causing a problem for a 
business, and the identification of the cause is accomplished by comparing similar 
situations with and without problems. Problems with multiple solutions include the 
problems faced by mission planners, who use the general method of generating 
several options (COAs) to be evaluated. 
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Figure 8. Expansion of the model for the metacognitive skills abstracted as 
Problem Representation and Problem Solving. 

The application of the problem-solving method selected, as in the performance 
of any task, requires both task knowledge (declarative knowledge) and task skills 
(procedural knowledge). Similarly, the performance of the final three tasks in the 
problem-solving sequence — implementation, monitoring, and evaluation of the 
solution — requires both procedural and declarative knowledge, in the sense that Reif 
and Larkin's physics students needed to know not only the physics principle to be 
applied to a problem but also how to apply it (procedural knowledge). Figure 9 depicts 
these last three skills, and we must add allocation of resources, in this case for the 
implementation phase. For example, a military commander makes an array of forces 
in his mission planning and also allocates other resources such as helicopters and 
artillery. 
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Figure 9. Metacognitive skills in implementation, monitoring, and evaluation. 

TEMPORAL, SOCIAL, AND ORGANIZATIONAL MONITORING AND CONTROL 

Technical skills, cognitive or metacognitive, are better conceptualized than 
temporal, social, and organizational skills, although the latter are of critical importance 
in the assessment and training of executive performance. What we present in this 
section is by no means an adequate model of these nontechnical skills, but rather a 
scaffolding for the later construction of such a model. We have some comments on 
what these models might look like, at the end of this section. 

Temporal skills. As defined previously, temporal skills have to do with the 
effective use of time in problem solving. Scheduling is one such skill, one that enables 
the executive to allocate temporal resources effectively to complete the task in the 
allotted time. For example, the Army has developed the “1/3, 2/3 rule” for mission 
planning, that is, a commander at any echelon should use 1/3 of the total time before 
the beginning of mission execution for his planning and leave 2/3 for 
subordinate commanders to do their planning. Another important temporal skill is 
synchronization (Long, 1989). In Army practice, tactical commanders are taught to 
construct a synchronization matrix (Tactical Commanders Development Course at Fort 
Leavenworth), in which the temporal aspects of the actions (start, stop, etc.) of each of 
the seven Battlefield Operating Systems (BOS) are charted, to ensure maximum 
impact of the operation as a whole. In most complex problem solving, several 
resources must be synchronized. 

Social skills. Effective leaders allocate human resources wisely and 
delegate responsibility in a way that satisfies the technical, temporal, and 
organizational requirements of the task. Studies of leaderless groups show that two 
types of leaders typically emerge: a task leader, who facilitates the technical aspects of 
problem solving in a group, and a socioemotional leader, who facilitates the social 
aspects. There are several social skills relevant to problem solving. One is the ability 
to motivate subordinates and to use rewards (and possibly punishments) effectively. 
Laskey et al. (1990) speak of the importance of shared ownership, that is, an effective 
leader takes all points of view into account and develops a consensus in which all 
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participants consider themselves to have contributed to. Conflict management is 
another important social skill. 

Organizational skills. In a previous section, we referred to organizational 
skills as executive monitoring. Executives have key positions in a hierarchical network 
and, to be effective, must understand their position and its relationship with 
superordinates, subordinates, and peers. Executives must know the long-term goals 
of the organization and how such goals are achieved in the context of the 
organizational structure. In the business world, this skill is often described as "seeing 
the big picture" (Carducci, 1993). The big picture includes an understanding of how 
the department in which one works, say R&D, fits with other divisions of the company, 
such as marketing, sales, planning, accounting, etc. 

Executives are especially skilled at organizational development, which in the 
case of business organizations means business development; in the Army, 
organizational development means the creation, equipping, training, and fielding of 
military units, while exploring new technologies, training methodologies, and other 
innovations that might lead to more effective armies in the future. 

The conceptual model. Temporal, social, and organizational skills are 
clearly important to effective executive performance, but they have rarely been 
modeled. Even more rarely have these metacognitive skills been modeled in the 
same context as technical problem-solving skills. One possibility, which we will 
explore, is a generic model like the one presented in Figure 4 for technical problem 
solving. Perhaps the same model can be used for the four different sets of 
metacognitive skills: technical, temporal, social, and organizational. For temporal 
skills, the problems would most likely have a time line — “We are going to run out of 
ammunition around 1600 hours." The solutions would also be time-based; in many 
cases, they would be scheduling solutions or synchronization solutions. Allocation of 
resources over time would also contribute to the solutions. 

Similarly, the social skills would aid in the solution of social problems that are 
preventing the executive from reaching organizational goals. The technical problem 
may have been solved, but the executive’s subordinates are reluctant to execute the 
solution because of fear or fatigue. The social solutions would be of the sort that 
reduce conflict in the group and motivate the group members to work toward the 
group’s goals. Organizational skills would aid in the solution of organizational 
problems, e.g., if the organizational structure is such that no one has responsibility for 
certain subtasks, an organizational solution might assign such responsibility in an ad 
hoc fashion, to facilitate goal attainment. I am reminded of my boss on a road crew I 
once worked for: Working long hours in the hot summer, the workers complained of 
boredom and lack of motivation; they proposed that jobs be rotated, so that one day I 
might clean the road in advance of the seal-coating unit, the next day I might drive a 
packer, and the third day I might drive a gravel truck. A social motivational problem. 

The boss said, however, that the proposed solution could not be implemented 
because of a conflicting organizational problem that would arise if it were: No one 
worker would have responsibility for the maintenance of his piece of equipment, which 
we had to admit meant that the equipment would surely fall into disrepair. 
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TRAINING AND ASSESSMENT 


The purpose of the conceptual model is to guide training and assessment of 
metacognitive skills in executive-level commanders. To do so, the model must include 
training and assessment agents and objects derived from and related to the model of 
the metacognitively influenced task; the model must have training and assessment 
modules. This section will suggest possible answers to the questions of how we can 
train metacognitive skills and how we can assess them in experienced commanders. 

We are wandering into uncharted territory here, with only a few good pioneers 
to guide us. I warn you of this, to warn you that you will see less careful documentation 
of my ideas from here on in. We are in an early stage of science, one characterized by 
"mucking around" in the domain, as one philosopher once termed it, getting our feet 
wet, glad to see any solid stands of ground. 

TRAINING 

Most of the psychological research on metacognition has heretofore been 
developmental, focused on childhood changes with age and experience. This 
research gives some clues as to the ways such skills can be trained, but unfortunately 
the focus has been on when (at what age) such skills develop and what effect such 
skills have on children's performance (enhanced) in intellectual tasks. Case (1984), 
for example, describes ten levels of intellectual skill development, the last three or four 
of which are skills involving abstractions similar to the metacognitive processes 
described in the conceptual model. These skills presumably develop in the teens, with 
the ability to do “abstract mapping” observed in most cases between 14 and 16 years. 
Case says little about how these skills develop — some may require neurological 
development. Case does hypothesize that the limited processing capacity of humans 
is first devoted to basic operations and then, as these (cognitive) skills develop and 
require less conscious control, more of the capacity can be devoted to metacognitive, 
support skills. Sternberg (1984) has similar ideas, describing a process in which 
crude metacognitive skills (metacomponents) are used to control intellectual 
operations, receive and interpret feedback on the results of such operations, and 
refine themselves on the basis of that feedback. In Sternberg’s theory, “the 
metacomponents form the major basis for the development of intelligence” through 
continual feedback loops (Sternberg, 1984, p. 172). If the metacomponents are not 
used to increase metacognitive skills, significant increases in intellectual performance 
are unlikely; mere experience or practice will not be effective. 

Metacognitive skills support problem solving performance in abstract ways, 
providing a general framework and a general procedure. Thus, the problem solving 
methods described by Kepner and Tregoe (1965) and Plunkett and Hale (1982) can 
be taught as metacognitive skills. In such training programs, students are taught how 
to detect, define, and describe the problem, and then how to identify and eliminate the 
cause of the problem. These training programs, in a sense, teach the scientific method 
for problem solving. 

The science training program of Reif and Larkin adds the substance to the 
promise. Reif and Larkin have, in effect, trained metacognitive skills in problem 
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solving for the domain of academic physics courses, with astounding success. We will 
borrow heavily from Reif and Larkin in our training program design. We need only 
apply the techniques to a new domain, Army command and control. 

It is important to stress the recurring note in these investigations: Unless one 
teaches the procedural knowledge, the semantic knowledge will be 
useless. It is imperative to tell your students how to do it, that is, the nitty gritty, the 
hands-on, no holds barred, this is step one, this is step two, etc. 

Can one train people to use abstract, domain-independent inferential rules to 
think about important events in their lives? A surprising number of theorists say no, 
that Plato’s doctrine of formal discipline, which holds that the study of abstract rule 
systems trains the mind for reasoning about concrete problems, is invalid (Thorndike, 
1906). Thorndike was able to show that there was very little transfer of training from 
one course of study (e.g., Latin) to other courses. If his view is correct, we will have 
little luck training metacognitive skills. But current work on this issue, exemplified by 
the research of Nisbett and his colleagues (Nisbett et al., 1987), suggests that abstract 
skills can be taught. The primary problem in the transfer of training is in the ability of 
students to apply the abstract rules to specific domain content. This, the proper 
representation of the problem so that the abstract rule can be seen to apply, is what 
needs to be taught. 

Specific metacognitive skills have been the substance of specific training 
courses, and experience of this sort also illuminates our approach to training 
metacognitive skills in general. For example, synchronization of resources has been 
trained at Fort Leavenworth for several years, in a precommand course called Tactical 
Commander’s Development Course (TCDC). Students are taught a general 
methodology that can be applied to specific mission-planning exercises (Long, 1989). 
In essence, they are given a matrix to plan the activities of each of the seven Battle 
Operating Systems (BOSs) along a time line that begins before H-hour and continues 
into sequel missions. They are given extensive practice filling in the matrix in a variety 
of mission-planning exercises and encouraged to continue the practice in their 
command roles. Synchronization is a metacognitive skill related to temporal 
monitoring. 

Social monitoring and control comprises a set of metacognitive skills related to 
team or group performance. Salas and his colleagues have been developing theory 
concerned with the training of teamwork skills, which has obvious implications for 
training metacognitive skills (e.g., Glickman et al., 1987). Findings include the fact that, 
for most team training, training in a team context is superior to individual training on 
team tasks. Also, three distinct factors appear in team training: a taskwork factor, 
reflecting training on the team task; a teamwork factor, reflecting learning to coordinate 
and communicate within the team; and a jelling factor, which reflects the ability to put 
the taskwork and the teamwork together in an integrated approach to problem solving 
in the group setting. These three factors are distinct at the beginning of training, but 
converge during the final stages of training. These findings bear a striking 
resemblance to the concept of two kinds of leadership: task leadership and 
socioemotional leadership. 
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ASSESSMENT 


The first step in skill assessment is a thorough task analysis, according to the 
standard Instructional Systems Development (ISD) methodology (Vineberg & Joyner, 
1980). If the goal is to assess a cognitive skill, one must do a cognitive task analysis. 
Our goal is to assess a metacognitive skill; what then is required of us? A 
metacognitive task analysis? Procedural or behavioral task analysis is a fairly well- 
defined technique (Drury et al., 1987), and even cognitive task analysis is becoming 
more common and more standardized (Lesgold et al., 1990). Methods for group 
cognitive task analysis have been developed (Salas, 1993). But I know of no work on 
individual or group metacognitive task analysis. Before we can begin to construct tests 
of metacognitive skills, we must first develop such a task-analysis methodology. The 
training of metacognitive skills also depends on such analyses. 

Once the appropriate task analysis has been accomplished, the knowledge and 
skills (KSs) required to perform each step are determined. The KSs (not the 
procedural steps of the task) are the "raw material" of both training programs and 
assessment devices. Classroom training is designed to provide the task knowledge, 
whereas laboratory and on-the-job training is designed to teach task skills. Verbal 
tests are designed to assess task knowledge and performance tests are designed to 
assess task skills. So the question before us is, How do we assess metacognitive 
skills and knowledge related to problem solving? 

Cognitive task analysis uses knowledge-acquisition techniques (KATs) to elicit 
the knowledge and the covert decision processes involved in cognitive task 
performance (Geiwitz, Kornell, & M’Closkey, 1992). Many KATs also seem 
appropriate for the investigation of metacognitive KSs. Protocol Analysis, for example, 
has a domain expert “think out loud” while performing the task (Ericsson & Simon, 
1984). If the expert were primed, not to describe the direct problem-solving processes, 
but to describe the goals of the endeavor and the strategy for goal attainment, we 
might elicit the metacognitive steps that monitor and control the direct processes. 

These steps could then be analyzed for required KSs in a conventional manner. 
Similarly, since effective problem representation is a key metacognitive skill, we could 
use the KAT called Cognitive Structure Analysis (Leddo & Cohen, 1988). (This 
technique grows from a conceptual model called Integrated Knowledge Structures — 
INKS — described by Laskey et al., 1990.) Cognitive Structure Analysis purports to 
identify the knowledge representations the expert uses: production rules, scripts, 
frames, semantic networks, or mental models. Not only would this KAT be useful in 
identifying the expert’s representation of the problem, it might also describe the 
expert’s overarching representation of the problem-solving process. I suspect that 
most scientists have a mental model of the scientific method that they use for the 
everyday conduct of scientific activity. 

Several metacognitive skills are the subject of psychological research, and the 
criterion variables used to represent these skills may provide a means of assessment. 
Aircrew coordination and communication, for example, has been operationalized as 
ratings based on specific behaviors, e.g., the discussion of potential coordination 
problems during preflight briefings (Franz et al., 1990). Performance in games or 
simulations has also been used to define metacognitive skills. These games require 
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coordination between two or more team members for superior performance (Bowers et 
al., 1992). 

Finally, there are numerous tests available for the assessment of reasoning and 
problem solving ability. Tests of diagnostic ability (e.g., troubleshooting) are also 
available. 

We should mention a special assessment technique known as Career Path 
Appreciation (CPA; Stamp, 1988). CPA was developed to measure the level of 
cognitive complexity that a member of an organization was dealing with at the present 
time. This level is assumed to predict later career development according to Stratified 
Systems Theory (Jacobs & Jaques, 1987). CPA is essentially a structured interview 
that focuses on a respondent’s general approach to problems; it should therefore be 
well adapted for the assessment of metacognitive skills. However, very little 
information is available on the technique or on the scoring of the interview protocols. 

TRAINING METACOGNITIVE SKILLS 

Table 2 aligns the ten metacognitive skills with the procedural steps for 
mission planning known as the Command Estimate. The tables following 
describe in outline form the nature of the training to be developed in the Army 
domain. 
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TABLE 2 


STEPS IN THE COMMAND ESTIMATE AND 
CORRESPONDING METACOGNITIVE SKILLS 


THE COMMAND ESTIMATE 
Receive Mission 
Analyze Mission..»~^..~..~~~~~. 
Restate Mission— 

CDR's Guidance.....~..~..MM~~~~. 

Develop COAs..m.....~^~....~~~~~« 

^^nalyze 

Recommend 

Select COA— 


METACOGNITIVE SKILLS 
Detect Problem 
Represent Problem 
Represent Problem 
Plan Strategy for Problem Solving 
Apply Problem>Solving Method 
Apply Problem-Solving Method 
Describe Solution 
Describe Solution 


Construct OPORD. 


.. Communicate Solution 











TABLE 3 


THE COMMAND ESTIMATE 

Detection of the problem: No training. Mission comes 
from higher unit. 

Representation of the problem: the Intelligence 
Preparation of the Battlefield (IPB), very detailed, very 
procedural. 

Selection of ps method: always use the same method 
(the command estimate process) 

Resource allocation: very little not specified by 
doctrine, e.g., G2 staff responsible for IPB. 

Strategic application of ps method: always perform the 
same. 

Evaluation of solution candidates: three COAs and the 
decision matrix; wargaming the COAs 

Recognition of errors: no procedures 

Temporal monitoring: synchronization matrix 

Social monitoring: trained as teams 

Organizational monitoring: the military's big 
advantage; everyone knows the command structure 
and how it works 
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TABLE 4 


SPECIFIC DEFICIENCIES IN 
THE COMMAND ESTIMATE 

Representation of the problem in terms of goals: 
Planners must know the commander's intent two ieveis 
up, but make no reai use of this information. 

Commander's Guidance: no reai use now. cdrs often 
say nothing, to see what their staff can come up with, 
then they say, none of these, this instead, staff feels 
like mushrooms, kept in dark and fed shit. Wiii train 
cdrs to identify "Dangers and Opportunities" in the 
situation, teii staff to worry about dangers, expioit 
opportunities. 

Probiem-soiving method: is one sufficient? decision 
matrix needs work, e.g., no weighting of factors. 

Seiection of COA: If cdr does not go with staff 
recommendation, then the COA chosen is never 
evaiuated. Aiso true if a combination of COAs is 
chosen. 
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TABLE 5 


TRAINING METACOGNITIVE SKILLS IN 
THE COMMAND ESTIMATE 

Current training aiready trains severai metacognitive 
skiils to aid severai of the steps. Need to highlight 
metacognitive aspects, provide more deciarative 
knowiedge, and ensure that the procedurai knowiedge 
is experienced and practiced. 

ISD methodoiogy: Thorough task anaiysis and 
appropriate knowledge acquisition ~> knowiedge and 
skilis for task performance -> metacog knowiedge and 
skills for superior task performance. 

Teach metacog knowledge in lectures, train metacog 
skilis in iaboratory exercises. Provide iecture notes 
and overheads. Provide procedures and worksheets 
for laboratory exercises, with quizzes. 

Assess metacog knowledge thru verbal (multiple 
choice) tests; assess metacog skilis thru objective 
performance tests. 
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TABLE 6 


TRAINING: 

PROBLEM DETECTION 


COMMAND ESTIMATE 

RECEIVE MISSION 
basic question 

declarative knowledge 

procedural knowledge 


METACOGNITIVE 
SKILLS OR 
KNOWLEDGE 

PROBLEM DETECTION 

How do you know a 
problem exists? 

--compare current state to 
goal state, discrepancy 
indicates problem 
-military situation: problem 
detection = reception of 
orders 

-represent problem in 
current state, use same 
representation for goal 
state; note discrepancies for 
goals and subgoals 
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TABLE 7 


TRAINING: 

PROBLEM REPRESENTATION 


COMMAND ESTIMATE 

METACOGNITIVE 
SKILLS OR 

ANALYZE MISSION, 

RESTATE MISSION 

KNOWLEDGE 

PROBLEM 

REPRESENTATION 

basic question 

How should the problem be 
described? 

declarative knowledge 

-goals most important, must 
be stated explicitly, for the 
unit and for two levels up 
(commander's Intent) 
“Importance: a) sets the 
goal state; b) if cut off, can 
continue mission 
-represent problem in terms 
of battlefield objects, 
including terrain, and the 
forces on each object (size 
and strength of units, 
friendly and enemy) 

procedural knowledge 

“determine facts and 
assumptions 
“G2 applies IPB 
-mission analysis (goals + 
specified & implied tasks) 


-constraints & restrictions 
“time analysis 
-develop SitMap using 
military symbols 
-overlays (COO, situation, 
event, decision support) 
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TABLE 8 


TRAINING: 

COMMANDER'S GUIDANCE 


METACOGNITIVE 



COMMANDER'S PLAN STRATEGY FOR 

GUIDANCE PROBLEM SOLVING 

basic question How should the planners 

proceed? 

declarative knowledge —Dangers are aspects of the 

situation that present 
planning difficulties; 

- Opportunities are aspects 
of the situation in which the 
relative advantage falls to 
us 

-combat power ratios: 1:3 
for defense, 3:1 for attack 

procedural knowledge 

-present mission and 
restated mission 
-incorporate time factors 
-state acceptable risk 
-Dangers & Opportunities 
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TABLE 9 


TRAINING: 
DEVELOP COAS 


COMMAND ESTIMATE 

METACOGNITIVE 

SKILLS OR 

DEVELOP COURSES OF 

ACTION 

KNOWLEDGE 

APPLY PROBLEM¬ 
SOLVING METHOD 

basic question 

What are the best courses of 
action? (usually best three) 

declarative knowledge 

-goals and tasks from the 
restated mission 
-combat power ratios 
-maneuver schemes 
-command & control 
requirements 

procedural knowledge 

1. analyze relative combat 
power 

2. array initial forces 

3. develop scheme of 
maneuver 

4. determine C2, control 
measures 

5. prepare COA statements 
& sketches 
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TABLE 10 


TRAINING: 
ANALYZE COAS 


METACOGNITIVE 



EVALUATE COURSES OF APPLY PROBLEM- 

ACTION SOLVING METHOD 


basic question 
declarative knowledge 


procedural knowledge 


Which of the 3 COAs is 
best? 

“Wargaming theory, 
including enemy doctrine 
(G2) 

-criteria for decision matrix 
-criteria weights 
-7 BOSS analyzed 
-G1, G4 wargame too 

-wargaming procedure 
-decision matrix 
-temporal monitoring: 
synchronization matrix 
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TABLE 11 


TRAINING: 

RECOMMEND, CHOOSE, COMMUNICATE 

COAS 


COMMAND ESTIMATE 

METACOGNITIVE 
SKILLS OR 

RECOMMEND, CHOOSE, 
COMMUNICATE COA 

KNOWLEDGE 

APPLY PROBLEM¬ 
SOLVING METHOD: 
DESCRIBE SOLUTION: 
COMMUNICATE 

SOLUTION 

basic question 

Which COA should we 
follow? 

declarative knowledge 

-decision matrix 
“Army orders formats 
-command & control 
requirements 

procedural knowledge 

-evaluate solution (ala Reif) 
-make decision 
“describe in terms of 
problem 

-describe in terms of orders 
formats 
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APPENDIX A 




INSTRUCTIONS FOR PROBLEM DESCRIPTION 

The following are detailed instructions for effective problem description, used in Heller 
and Reif (1984). Two models are included: M is the metacognitive skill model, and M* 
is the conventional model for description taught in most physics textbooks. Instruction 
not specifically marked are common to both models. E=experimenter, S=subject. 


Theoretical Description of Systems 

M PE: “Let’s now draw diagrams describing each system of 
L interest.” 

M* r E: “Let’s now draw diagrams describing the forces on 
L each system of interest.” 


Choice of. Particular System 


E: 

“Which system . . .do you wish to consider 
(first)/(next)?” 

(1) 

S: 

Names a system ‘X’. 


If X is a string or is not affected by interactions 
with other systems: 



E: “There is no need to describe X.'* 

Return to step 1. 

(2) 

Else continue: 



M Motion Description 
only 

E: “First draw a motion diagram of X, including any (3) 

available information about its position, velocity, 
and acceleration relative to a convenient reference 
frame. If the velocity or'acceleration is zero, 
indicate that on your diagram.” 

S; Draws motion diagram of X. 

E: “It is also useful to include on this diagram any (4) 

known properties of the system, such as mass.” 

If previous systems have been described: 

E: “Be sure to use convenient symbols and to relate (5) 
them to those you’ve used previously.” 

If X has circular motion: 

E: “Remember, the acceleration of a system in (6) 

circular motion ordinarily, although not 
always, has two components, one tangential and 
the other toward the center of the circle. 

Check to be sure whether both components 
exist in this case.” 


A-1 


Interaction Description 

E: “Now let’s draw an interaction diagram tor a, 
using the method I’ve suggested.” 

Short-Range Forces 

w E: “First name each system that touches A, 

including those that exert applied forces. 
As you identify each system, indicate all 
external contact forces exerted on X by 
that system.” 

M* r E: “Draw a force diagram indicating the forces 

L exerted on X by all other systems.” 

jy previous systems have been described: 

E: “Be sure to use convenient symbols 
and to relate them to those you’ve 
used previously.” 

S; Names interacting systems (‘1") and/or 
indicates forces. 

M r ^interaction with surface: 
only 

E: “Remember, the force exerted by a 

surface ordinarily, although not always, 
has two components, the normal force 
and friction force. Check to be sure 
whether both components exist in this 
case. 

The normal force is perpendicular to 
the surface and directed away from it. 

The friction force opposes the relative 
motion of the contact points—here it 
opposes the motion of X relative to 

y.” 

Long-Range Forces 

E: “ Name all external systems that directly 
interact with X without touching it or 
through any other physical contact. Then 
indicate the long-range forces exerted on 
X by each such system.” 

S: Names system and/or indicates force. 

Check: Missing or Extraneous Forces 
M [ E: “Are there any other systems touching .AT?” 

M* r E: “Are there any other forces on X by 

L anything else? 

S: “Yes” or “no”. 


(7) 

(7') 

( 8 ) 

(9) 

( 10 ) 

( 11 ) 

( 12 ) 

( 12 ') 
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If yes: 

E: “Draw the forces exerted by that (those) 
system(s)." 

E; “Draw the forces.” 

Return to step 12. 

Else continue: 

E: “Are there any other systems directly interacting 
with X by long-range forces?” 

S: “Yes" or “no”. 

If yes: 

E: “Draw the force exerted by that system.” 
Return to step 14. 

Else continue: 

E: “If not, you are finished describing all 
forces on X. Do not add any others. 

Check: Consistency Between Motion and Interaction 

E: “The motion and interaction of the system must be 
consistent. In your diagrams, are the forces on X 
such that, with proper magnitudes, their vector sum 
can have the same direction as X's acceleration? 
Show me how you determine this. (You might want 
to check whether this is true by comparing 
components along convenient directions.)” 

S: Checks consistency; responds “yes” or “no” with 
explanation. Modifies description(s) if necessary. 

E: “What would have to be true about the relative 
magnitudes of the forces on X for the acceleration 
and resultant force to have the same direction?” 

S: Describes required relative magnitudes of forces. 

Repetition of Description for Each System 

E: “Have all systems of interest been described yet?” 

S; “Yes” or “no”. 


M 

M* 


M 
only 


Repeat theoretical description procedure, beginning (20) 

at step 1. 

Else continue: 

Check of Entire Description 

E: “After describing all systems, it’s useful to double-check 
your work. Let’s run through a checklist to make sure 
you haven’t missed anything.’’ 

Check: Choice of Useful Symbols 

E: “All arrows should be labeled.’’ (21) 

S: Checks arrows. 

E: “Except for the gravitational force (which may (22) 

be expressed as “mg”), or any magnitudes actually 
given in the problem statement, the values of 
quantities should not be evaluated at this time. 

Symbols like “F,” “T,” and “N,” with subscripts, 
should be used instead.” 

S: Checks symbols. 

E: “Look at the symbols in all of your diagrams. (23) 

Wherever different symbols have been used, the 
values of these quantities should actually be 
unrelated. If values are the same or simple 
multiples, use the same symbol. If values are 
unrelated, different symbols should be used.” 

S: Checks symbols. 

Check: Use of all Information in Problem 

E: “All information specified in the problem should (24) 
be incorporated in your analysis. Please reread 
the problem carefully to make sure you have 
considered all the given information. In particular, 
make sure you’ve obtained from the problem all 
available information about the magnitude and 
direction of the velocity and acceleration of 
each system.” 

S: Rereads problem statement. Modifies descriptions 
if needed. 
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Check: Exploitation of Constraints (Mutual Forces) 

E: "Check to make sure that all action-reaction (25) 

pairs of forces are described as equal in magnitude 
and opposite in direction. For example, if systems 
A and B interact, the force o( A on B in your 
diagram of B should be opposite in direction but 
should have the same magnitude as the force of B 
on A in your diagram of A. Look for forces 
between each pair of systems and check that they 
are described right.” 


S: Checks forces. 
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